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Abstract 

We show how to efficiently simulate the sending of a message M to a receiver who has 
partial information about the message, so that the expected number of bits communicated in 
the simulation is close to the amount of additional information that the message reveals to the 
receiver. This is a generalization and strengthening of the Slepian-Wolf theorem, which shows 
how to carry out such a simulation with low amortized communication in the case that M is a 
deterministic function of X. A caveat is that our simulation is interactive. 

As a consequence, we prove that the internal information cost (namely the information re- 
vealed to the parties) involved in computing any relation or function using a two party interactive 
protocol is exactly equal to the amortized communication complexity of computing independent 
copies of the same relation or function. We also show that the only way to prove a strong direct 
sum theorem for randomized communication complexity is by solving a particular variant of the 
pointer jumping problem that we define. Our work implies that a strong direct sum theorem for 
communication complexity holds if and only if efficient compression of communication protocols 
is possible. 
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1 Introduction 



Suppose a sender wants to transmit a message M that is correlated with an input X to a receiver 
that has some information Y about X. What is the best way to carry out the communication 
in order to minimize the expected number of bits transmitted? A natural lower bound for this 
problem is the mutual information between the message and X, given Y: I{M;X\Y), i.e. the 
amount of new information M reveals to the receiver about X. In this work, we give an interactive 
protocol that has the same effect as sending M, yet the expected number of bits communicated 
is asymptotically close to optimal — it is the same as the amount of new information that the 
receiving party learns from M, up to a sublinear additive terrr0. 

Our result is a generalization of classical data compression, where Y is empty (or constant), 
and M is a deterministic function of X. In this case, the information learnt by the receiver is 
equal to the entropy H{M), and the compression result above corresponds to classical results on 
data compression first considered by Shannon |Sha48| — M can be encoded so that the expected 
number of bits required to transmit M is H{M) + 1 (see for example the text |CT91j ). 

Typical work in information theory usually focuses on the easier problem of communicating 
n independent copies Mi, . . . ,M„, where each Mi has an associated dependent Xi,Yi. Here n is 
viewed as a growing parameter, and the average communication is measured. Indeed, any solution 
simulating a single message can be applied to simulate the transmission of n messages, but there is 
no clear way to use an asymptotically good solution to compress a single message. By the asymptotic 
equipartition property of the entropy function, taking independent copies essentially forces most 
of the probability mass of the distributions to be concentrated on sets of the "right" size, which 
simplifies this kind of problem significantly. The Slepian-Wolf theorem |SW73j addresses the case 
when M is determined by X. The theorem states that there is a way to encode many independent 
copies Ml, . . . , Mn using roughly /(M; X|y) on average, as n tends to infinity. The theorem and 
its proof do not immediately give any result for communicating a single message. Other work has 
focused on the problem of generating two correlated random variables with minimal communication 
|Cuf08j ■ and understanding the minimal amount of information needed to break the dependence 
between X,Y |Wyn75| , neither of which seem useful to the problem we are interested in here. 

Motivated by questions in computer science, prior works have considered the problem of en- 
coding a single message where M is not necessarily determined by X (see |JRS03t IHJMR07] and 
the references there), but these works do not handle the case above, where the receiver has some 
partial information about the sender's message. 

2 Consequences in Communication Complexity 

Given a function f{x,y), and a distribution fi on inputs to /, there are several ways to measure 
the complexity of a communication protocol that computes /. 

• The communication complexity Dp, namely the maximum number of bits communicated by 
a protocol that computes / correctly except with probability p. 

^Observe that if X,Y,M are arbitrary random variables, and the two parties are tasked with sampling A/ effi- 
ciently (as opposed to one party transmitting and the other receiving), it is impossible to succeed in communication 
comparable to the information revealed by A'l. For example, if M = f{X, Y), where / is a boolean function with high 
communication complexity on average for X, Y , M reveals only one bit of information about the inputs, yet cannot 
be cheaply sampled. 
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• The amortized communication complexity, lim.„^oo -Dp'""/?!, where here Dp'^ denotes the 
communication involved in the best protocol that computes f on n independent pairs of 
inputs drawn from fi, getting the answer correct except with probability p in each coordinate. 

Let Tr{X, Y) denote the public randomness and messages exchanged when the protocol vr is run 
with inputs X, Y drawn from fj,. Another set of measures arises when one considers exactly how 
much information is revealed by a protocol that computes /. 

• The minimum amount of information that must be learnt about the inputs by an observer 
who watches an execution of any protocol {I{XY;7r{X,Y))) that compute / except with 
probability of failure p, called the external information cost in |BBCR10] . 

• The minimum amount of new information that the parties learn about each others input by 
executing any protocol {I{X;Tr{X,Y)\y) + I(Y;'k{X,Y)\X)) that computes / except with 
probability of failure p, called the internal information cost in |BBCR10] . In this paper we 
denote this quantity \Cp{f,p). 

• The amortized versions of the above measures, namely the average external/internal infor- 
mation cost of a protocol that computes f on n independent inputs correctly except with 
probability p in each coordinate. 

Determining the exact relationship between the amortized communication complexity and the 
communication complexity of the function is usually referred to as the direct sum problem, which 
has been the focus of much work [CSWYOll ISha03l IJRS031 IHJMR071 IBBCRlOl IKlalOj . For 
randomized and average case complexity, we know that n copies must take approximately (at 
least) ^/n times the communication of one copy, as shown by the authors with Barak and Chen 
|BBCR10] . For worst case (deterministic) communication complexity, Feder, Kushilevitz, Naor, 
and Nisan |FKNN9T] showed that if a single copy of a function / requires C bits of communication, 
then n copies require Q{^/Cn) bits. In the rest of the discussion in this paper, we focus on the 
average case and randomized communication complexity. 

The proofs of the results above for randomized communication complexity have a lot to do with 
the information theory based measures for the complexity of communication protocols. Chakrabarti, 
Shi, Wirth and Yao [CSWYOl] were the first to define the external information cost, and prove 
that if the inputs are independent in p, then the external information cost of / is at most the 
amortized communication complexity of /. This sparked an effort to relate the amortized commu- 
nication complexity to the communication complexity. If one could compress any protocol so that 
the communication in it is bounded by the external information cost, then, at least for product 
distributions p, one would show that the two measures of communication complexity are the same. 

For the case of general distributions p, it was shown in |BYJKS04l IBBCRIO] that the amortized 
communication complexity can only be larger than the internal information cost. In fact, the 
internal and external information costs are the same when ^ is a product distribution, so the 
internal information cost appears to be the appropriate measure for this purpose. [BBCRIO] gave 
a way to compress protocols so that the communication is reduced to the geometric mean of the 
internal information and the communication in the protocol, which gave the direct sum result 
discussed above. 

The main challenge that remains is to find a more efficient way to compress protocols whose 
internal information cost is small. Indeed, as we discuss below, in this paper we show that this 
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is essentially the only way to make progress on the direct sum question, in the sense that if some 
protocol cannot be compressed well, then it can be used to define a function whose amortized 
communication complexity is significantly smaller than its communication complexity. 



2.1 Our Results 
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Figure 1: The relationships between different measures of complexity for communication problems, 
with the new results highlighted. The present work collapses the upper-right triangle in the diagram, 
showing that amortized communication complexity is equal to the internal information cost of any 
functionality. We further show three equivalent characterizations that would lead to a collapse in 
the lower-left triangle: strong direct sum theorems, near-optimal protocol compression and solving 
the Correlated Pointer Jumping efficiently. 



Our main technical result is a way to compress one round protocols according to the internal 
information cost, which corresponds to the problem of efficiently communicating information when 
the receiver has some partial information, discussed in the introduction. In fact, we design a 
protocol that solves a harder problem, that we describe next. We give a way for two parties to 
efficiently sample from a distribution P that only one of them knows, by taking advantage of a 
distribution Q known only to the other. We obtain a protocol whose communication complexity 
can be bounded in terms of the informational divergence D {P\\Q) = P{x) \og{P{x)/Q{x)). 

Theorem 2.1. Suppose that player A is given a distribution (described by the probabilities assigned 
to each point) P and player B is given a distribution Q over a universe U. There is a public coin 
protocol that uses an expected 

D (P| IQ) + 2 log(l/e) + O (v/D(P||Q) + l) 

bits of communication such that at the end of the protocol: 
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• Player A outputs an element a distributed according to P; 

• Player B outputs b such that for each x P[6 = x\ a = x] > 1 — e. 

As a corollary, we obtain the formulation discussed earlier. For any distribution X, Y and 
message M that is independent of Y once X is fixed, we can have the sender set P to be the 
distribution of M conditioned on her input x, and the receiver set Q to be the distribution of 
M conditioned on her input y. The expected divergence D(P||(5) turns out to be equal to the 
mutual information I{M; X\Y). Indeed, applying Theorem 12.11 to each round of communication 
in a multiround protocol, gives the following corollary, where setting r = 1 gives the formulation 
discussed in the introduction. The proof appears in Section [6.11 

Corollary 2.2. Let X, Y be inputs to a k round communication protocol vr whose internal infor- 
mation cost is I. Then for every e > 0, there exists a protocol t such that at the end of the protocol, 
each party outputs a transcript for vr. Furthermore, there is an event G with P[G] > 1 — ke such 
that conditioned on G, the expected communication of t is I + 0{VkI + k) + 2/c log(l/e), and both 
parties output the same transcript distributed exactly according to 'ir(X,Y). 

This compression scheme significantly clarifies the relationship between the various measures 
of complexity discussed in the introduction. In particular, it allows us to prove that the internal 
information cost of computing a function / according to a fixed distribution is exactly equal to the 
amortized communication complexity of computing many copies of /. 

Theorem 2.3. For any f, fi, and p, 

\C,{f,p)= hm 

n— >oo n 

This result seems surprising to us, since it characterizes the information cost in terms of a 
quantity that at first seems to have no direct connection to information theory. The proof appears 
in Appendix 16.21 It proves that if a function's information cost is smaller than its communication 
complexity, then multiple copies of the function can be computed more efficiently in parallel than 
sequentially. Observe that the naive sequential protocol for computing multiple copies would only 
give a bound on the error in each copy separately (exactly as in our definition of amortized commu- 
nication complexity). The consequences to the various measures discussed earlier are summarized 
in Figured! 

In Appendix 16.31 we define a communication problem we call Correlated Pointer Jumping - 
CPJ(C, /) - that is parametrized by two parameters G and / such that G ^ I. CPJ(C, I) is 
designed in a way that the randomized communication complexity cost / < i?(CPJ(C, /)) < G. 
We show that determining the worst case randomized communication complexity i2(CPJ(C, /)) for 
I = G/n is equivalent (up to poly-logarithmic factors) to determining the best parameter k{n) for 
which a direct sum theorem R{f^) = 0,{k{n) ■ R{f)) holds. For simplicity, we formulate only part 
of the result here. 

Theorem 2.4. // R(CPJ(C, C/n)) = 0{G/n) for all G, then a near optimal direct sum theorem 
holds: R(/") = Vt{n ■ R(/)) for all /. 

On the other hand, if R(CPJ(C, C/??,)) = Vl{{G log"" G) / n) for all a > 0, then direct sum is 
violated by CPJ(C, C/n); 

R(CPJ(C,C/n)'^) = O(ClogC) = o(?i • R(CPJ(C, C7/n))/log" C), 

for all a. 
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Finally, letting /" denote the function that computes n copies of / on n different inputs, our 
protocol compression yields the following direct sum theorem: 

Corollary 2.5 (Direct Sum for Bounded Rounds). Let C be the communication complexity of the 
best protocol for computing f with error p on inputs drawn from fi. Then any r round protocol 
computing on the distribution with error p — £ must involve at least 0,{n{C — rlog(l/e) — 
0{y/C ■ r))) communication. 

2.2 Techniques 

The key technical contribution of our work is a sampling protocol that proves Theorem 12. li The 
sampling method we show is different from the "Correlated Sampling" technique used in work on 
parallel repetition |Hol07l IRaoOSj and in the previous paper on compression [BBCRIO] . In those 
contexts it was guaranteed that the input distributions P, Q are close in statistical distance. In 
this case, the sampling can be done without any communication. In our case, all interesting inputs 
P, Q are very far from each other in statistical distance, and not only that, but the ratios of the 
probabilities P{x)/Q{x) may vary greatly with the choice of x. It is impossible to solve this problem 
without communication, and we believe it is unlikely that it can be solved without interaction. 

Indeed, our sampling method involves interaction between the parties, and for good reasons. 
In the case that the sample is x for which P{x)/Q{x) is very large, one would expect that a lot 
of communication is needed to sample x, since the second party would be surprised with this 
sample, while if P{x)/Q{x) is small, then one would expect that a small amount of communication 
is sufficient. Our protocol operates in rounds, gradually increasing the number of bits that are 
communicated until the sample is correctly determined. 

To illustrate our construction, consider the baby case of the problem where the issue of high 
variance in P{x)/Q{x) does not affect us. Recall that the informational divergence D (P||Q) is equal 
to P{x) log ^|fy- Suppose Q is the uniform distribution on some subset Sq of the universe U, 
and P is the uniform distribution on some subset Sp C Sq. Then the informational divergence 
D(P||Q) is exactly log{\SQ\/\Sp\). 

In this case, the players use an infinite public random tape that samples an infinite sequence 
of elements ai,a2,... uniformly at random from the universe U. Player A then picks the first 
element x that lies in Sp to be his sample. Next the players use the public randomness to sample 
a sequence of uniformly random boolean functions on the universe. A then sends a stream of these 
functions evaluated at x. At each round i, player B finds the first element yi on the tape that 
belongs to Sq and is consistent with the values Player A has sent so far. Player B uses yi as his 
working hypothesis for the element Player A is trying to communicate. Player B lets Player A 
know (and outputs yi) if the element yi stays the same for some interval i = [j..2j + log That 
is, when the hypothesis for the element x stops changing. For the analysis, one has to note that 
the (expected) number of elements that B will have to reject before converging to x is bounded in 
terms of log |5'q|/|S'p| - the divergence between P and Q in this case. 

3 Preliminaries 

Notation. We reserve capital letters for random variables and distributions, calligraphic letters 
for sets, and small letters for elements of sets. Throughout this paper, we often use the notation \b 
to denote conditioning on the event B = b. Thus A|6 is shorthand for A\B = b. 
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We use the standard notion of statistical / total variation distance between two distributions. 

Definition 3.1. Let D and F be two random variables taking values in a set S. Their statistical 
distance is 

\D - F\ =^ max(| Pr[D G 71 - Pr[F eT]\) = -J2\ ^^t^ = •'^] " ^^i^ = 

If \D — F\ < e we shall say that D is e-close to F. We shall also use the notation D ^ F to mean 
D is £-close to F. 

3.1 Information Theory 

clef 

Definition 3.2 (Entropy). The entropy of a random variable X is H{X) = Pr[X = x] log(l/ Pr[X = 
x\). The conditional entropy H{X\Y) is defined to be 'Eiy&^Y \H{X\Y = y)]. 

Fact 3.3. H{AB) = H{A) + H{B\A). 

Definition 3.4 (Mutual Information). The mutual information between two random variables 
A,B, denoted I{A]B) is defined to be the quantity H{A) - H{A\B) = H{B) - H{B\A). The 
conditional mutual information I{A\B\C) is H[A\C) — H{A\BC). 

In analogy with the fact that H{AB) = H{A) + H{B\A), 

Proposition 3.5 (Chain Rule). Let Ci,C2, D, B be random variables. Then 

/(C7iC2; B\D) = I{Ci;B\D) + /(C2; B\CiD). 

We also use the notion of divergence (also known as Kullback-Leibler distance or relative en- 
tropy), which is a different way to measure the distance between two distributions: 

clef 

Definition 3.6 (Divergence). The informational divergence between two distributions is D (^| |i?) = 
E.A{x)log{A{x)/B{x)). 

For example, if B is the uniform distribution on {0, 1}" then D (y4||i?) = n — H{A). 

Proposition 3.7. Let A,B,C be random variables in the same probability space. For every a in 
the support of A and c in the support of C, let Ba denote B\A = a and B^c denote B\A = a,C = c. 
Then I{A; B\C) = Ea,c6,A,c [D {Bac\\B,)] 

Lemma 3.8. 

D (Pi X P2IIQ1 X Qa) = D (PillQi) + D (P2IIQ2) . 

3.2 Communication Complexity 

Let X, y denote the set of possible inputs to the two players, who we name A and B. In this papeiH, 
we view a private coins protocol for computing a function f : X x y ^ Zx as a rooted tree with 
the following structure: 

^The definitions we present here are equivalent to the classical definitions and are more convenient for our proofs. 
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• Each non-leaf node is owned by A or by B. 

• Each non-leaf node owned by a particular player has a set of children that are owned by the 
other player. Each of these children is labeled by a binary string, in such a way that this 
coding is prefix free: no child has a label that is a prefix of another child. 

• Every node is associated with a function mapping X to distributions on children of the node 
and a function mapping 3^ to distributions on children of the node. 

• The leaves of the protocol are labeled by output values. 
On input x, y, the protocol vr is executed as in Figure [2j 



Generic Communication Protocol 



1. Set V to be the root of the protocol tree. 

2. If is a leaf, the protocol ends and outputs the value in the label of v. Otherwise, the 
player owning v samples a child of v according to the distribution associated with her input 
for V and sends the label to indicate which child was sampled. 

3. Set V to be the newly sampled node and return to the previous step. 



Figure 2: A communication protocol. 

A public coin protocol is a distribution on private coins protocols, run by first using shared 
randomness to sample an index r and then running the corresponding private coin protocol vr^. 
Every private coin protocol is thus a public coin protocol. The protocol is called deterministic if 
all distributions labeling the nodes have support size 1. 

Definition 3.9. The communication complexity of a public coin protocol vr, denoted CC(7r), is the 
maximum number of bits that can be transmitted in any run of the protocol. 

Definition 3.10. The number of rounds of a public coin protocol is the maximum depth of the 
protocol tree vr^. over all choices of the public randomness. 

Given a protocol vr, 7r(x, y) denotes the concatenation of the public randomness with all the 
messages that are sent during the execution of vr. We call this the transcript of the protocol. 
We shall use the notation Tr{x, y)j to refer to the j'th transmitted message in the protocol. We 
write 7r{x, y)<j to denote the concatenation of the public randomness in the protocol with the first j 
message bits that were transmitted in the protocol. Given a transcript, or a prefix of the transcript, 
V, we write CC{v) to denote the number of message bits in v (i.e. the length of the communication). 

Definition 3.11 (Communication Complexity notation). For a function f : X x y ^ Z/<, a 
distribution /j, supported on X x y, and a parameter p > 0, Dpif) denotes the communication 
complexity of the cheapest deterministic protocol for computing / on inputs sampled according to 
/i with error p. Rp{f) denotes the cost of the best randomized public coin protocol for computing 
/ with error at most p on every input. 
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For ease of notations, we shall sometimes use the shorthand R{f) to denote Ri/^{f). 
We shall use the following theorem due to Yao: 

Theorem 3.12 (Yao's Min-Max). Rp{f) = max^L>^(/). 

Recall that the internal information cost IC'^(7r) of a protocol vr is defined to be I{tt{X, Y); X\Y)+ 
I{7r{X,Y)-Y\X). 

Lemma 3.13. Let R be the public randomness used in the protocol tt. Then IC'^(7r) = E/j [lC'^(7r/j)] 
Proof. By the chain rule (Proposition I3.5( ). 

ICV(7r) = I{7r{X, Y); X\Y) + I{Tr{X, Yy,Y\X) 

= I{R- X\Y) + I{R- Y\X) + /(7r(X, Y); X\YR) + /(7r(X, Y); Y\XR) 
= /(7r(X, y); X\YR) + /(^(X, Y)- Y\XR) 

= E [lC'^,(vr/?)] 

R 

□ 

A priori, one might believe that the internal information cost can be as large as twice the 
communication in a protocol. However, we can use the fact that each transmission only reveals 
information to one of the parties to bound it by the communication in the protocol: 

Lemma 3.14. \C ^{ti) < CC(7r). 

Proof. First, let us assume that vr is a private coin protocol. Let vTj denote the i'th bit transmitted 
in the protocol. Then, by the chain rule, 

ICV(vr) = /(7r(X, y); X\Y) + /(7r(X, Y)- Y\X) 

CC(7r) 

= ^ I{'Ki;X\-Ki'K2...-Ki-lY)+I{-Ki;Y\TTi'K2...T^i-lX) 
i=l 

Given any prefix 7 = vri . . . 7rj_i, let denote the event that the first i — 1 bits of the transcript 
are equal to 7. Then we have 

CC(7r) 

ICV(vr)= V E [I{7:,-X\E^Y)+I{tt,-Y\E.^X)]. 

~\ 7eR7ri...7r,_i 



Now we claim that l{T^i\X\E^Y^ + l(TXi\Y\E^X) < 1. Each of these terms is individually 
bounded by 1 since vTj contains only one bit. If 7 is such that it is the first party's turn to transmit 
TTj, then for every fixing of X, iTi is independent of Y, so /(vTj; Y\E.yX) = 0. On the other hand, if 7 
is such that it is the second party's turn to transmit vr,, then for every fixing of Y, VTj is independent 
of X, so I{TTi;X\E^Y) = 0. Thus, 
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CC(7r) 

ICV(vr) = V E X\ErY) + Y\ErX)] < CC(7r). 

-f— • rSp, 7ri...7ri_i 



If vr involves public randomness, then by Lemma 13.131 we have that IC'^(7r) = E/? [lC'^(7r^)] < 
CC(7r), where R denotes the public randomness of vr. 

□ 

A version of the following theorem was proved in |BYJKS04] . Here we need a slightly stronger 
version (alluded to in a remark in |BBCR10] ): 

Theorem 3.15. For every n, f, p there exists a protocol r computing f on inputs drawn from fi with 
probability of error at most p and communication at most Dp (/") such that IC'p(r) < ^ ^ — -. 

Since this theorem is subsumed by Theorem 13. 171 below, we do not give the details of its proof. 

For our results on amortized communication complexity, we need the following definition: we 
shall consider the problem of computing n copies of /, with error p in each coordinate of the 
computation, i.e. the computation must produce the correct result in any single coordinate with 
probability at least 1 — /o. We denote the communication complexity of this problem by Dp'"'{f) < 
D^"(/"). Formally, 

Definition 3.16. Let phe a distribution on X xY and let < p < 1. We denote by Dp'^{f) the 
distributional complexity of computing / on each of n independent pairs of inputs drawn from p, 
with probability of failure at most p on each of the inputs. 

The result above can actually be strengthened: 

Theorem 3.17. For every p, /, p, let tt be a protocol realizing Dp'"'{f). Then there exists a protocol 
T computing f on inputs drawn from p with probability of error at most p such that CC(t) = CC(7r) 

and \C'(t) < i^i^ < 

Proof. First let us assume that vr only uses private randomness. The protocol t{x, y) is defined as 
follows. 

1. The parties publicly sample J, a uniformly random element of the set {1, 2, 3, ... , n}. 

2. The parties publicly sample Xi, . . . , and Yj+i, . . . , Yn- 

3. The first party privately samples Xj+i, . . . ,Xn conditioned on the corresponding Y's. Simi- 
larly, the second party privately samples Yi, . . . , Ij-i. 

4. The parties set Xj = x, Yj = y, and run the protocol vr on inputs Xi, . . . , X^, ^i, . . . , Y^. 
They output the result computed for the J'th coordinate. 
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Observe that CC(t) = CC(7r), and the probabiUty of making an error in r is bounded by p. It 
only remains to bound IC'^(r) = I{X] t\Y) + I{Y] t\X). Let us bound the first term. 

I{X-T\Y)<I{X-TYi,...,Yn\Y) 

= I{X-JXi...Xj^iYi...Yn7:\Y) 

= I{X; JXi... Xj^iYi . . . Yn\Y) + I{Xj; 7t\JXi . . . Xj.^Yi . . . y„) 

= I{Xj-7T\JXi...Xj^lYi...Yn) 

where the final equality is from the fact that J, Xi, . . . , Yi, . . . ,Yn are all independent of 

X, Y, conditioned on every fixing of Y. 

Expanding the expectation according to J, we get by the Chain Rule: 

n 

I{X;t\Y) < {l/n)Y,HXj;7r\Xi...Xj^iYi...Yn) 

= /(Xi...X„;7r|yi...y„)/n 

Similarly, we can bound I{Y] t\X) < I{Yi . . . y„; 7r|Xi . . . Xn)/n, and thus IC'^(r) < IC'^j« < 
CC(7r)/n, by Lemma 13.141 

If TT uses public randomness -R, then denote by tr the protocol induced for each fixing of R. 
Then ICV(t) = Efl [lCV(rR)] < Eij [iC^n (7rR)/n] < CC(^)/n. 

□ 

4 Proof of Theorem 12.11 

We shall prove a stronger version of Theorem 12.11 

Theorem 4.1. Suppose that player A is given a distribution P and player B is given a distribution 
Q over a universe U. There is a protocol such that at the end of the protocol: 

• player A outputs an element a distributed according to P; 

• player B outputs an element b such that for each x, F[b = a\ a = x] > 1 — e. 

• the communication in the protocol is bounded by logP{a)/Q{a) + logl/e + loglogl/e + 
5^log P{a)/Q{a) + 9. 

Note that the second condition implies in particular that player B outputs an element b such that 
b = a with probability > 1 — e. The protocol requires no prior knowledge or assumptions on 
B{P\\Q). 

Proof. The protocol runs as follows. Both parties interpret the shared random tape as a sequence 
of uniformly selected elements {flij^i = {ixi,Pi)}°Zi from the set A := U x [0,1]. Denote the 
subset 

V := {{x,p) : P{x) > p} 
of A as the set of points under the histogram of the distribution P. Similarly, define 

Q := {{x,p) : Q{x) >p}. 
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For a constant C > 1 we will define the C-multiple of Q as 

C-Q:={{x,p)eA : {x,p/C) e Q}. 

We will also use a different part of the shared random tape to obtain a sequence of random 
hash functions hi : U ^ {0, 1} so that for any x ^ y £ U , F[hi{x) = hi{y)] = 1/2. 







Player A's side: 



Player B's side: 
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a2 


ai4 

ai2 


^13 






^15 


ai6 


an 


ay 


ag 

^4 aj 


a3 

aio 




aeX ag 


\ ag 










Figure 3: An illustration on the execution of the protocol. The elements Oj are selected uniformly 
from A = V( X [0, 1]. The first Oj to fall in V is uq, and thus player A outputs xq. Player A sends 
hashes of qq, which do not match the hashes of 05, the only a, in Q. Player B responds 'failure', 
and considers surviving elements in 2Q, which are and ag. After a few more hashes from A, 
is selected by B with high probability. 



We are now ready to present the protocol: 

1. Player A selects the first index i such that Oj = {xi,pi) £ V, and outputs xf, 

2. Player A uses 1 + [loglogl/e] bits to send Player B the binary encoding of k := [i/|Z//|] (if 
k is too large, Player A sends an arbitrary string); 

3. For all t, set parameters Ct := 2*^ st = 1 + [log 1/e] + (t + 1)^; 

4. Repeat, until Player B produces an output, beginning with iteration t = 0: 

(a) Player A sends the values of all hash functions hj{xi) for 1 < j < Sj, that have not 
previously been sent. 

(b) if there is an = {Vr-, Qr) with r £ {{k — 1) • + 1, . . . , k • \l/(\} in Ct • Q such that 
hjiVr) = hj{xi) for 1 < j < St, Player B responds 'success' and outputs y^,; if there is 
more than one such a,., player B selects the first one; 
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(c) otherwise, Player B responds 'failure', and the parties increment t and repeat. 

The output of Player A is distributed according to the distribution P, and further, the output is 
independent of k. To see this, note that the output is independent of whether or not k > s, for 
every s. 

Fix a choice of i and the pair {xi,pi) by Player A. Step H] of the protocol is guaranteed to ter- 
minate when > log P{xi)/Q{xi) since Oj belongs to q||^'| • Q. Denote T := ^log P{xi)/Q{xi)\. 
By iteration T, Player A will have sent st bits in Step HI and Player B will have sent T + 1 bits. 
Thus the amount of communication in Step H] is bounded by 

ST + T + 1 = 1 + [log l/e] + (T + 1)2 + T + 1 

< {^y\ogP{xi)/Qixi) + 2)2 + y'\ogPixi)/Qixi) + [logl/e] +3 



= log P(xi)/Q(xi) + 5v/logP(xO/g(xO + [log l/e] + 7, 
which shows that the total communication is at most 



log P{xi)/Q{x,) + 5^/log P{xi)/Q{xi) + log l/e + log log l/e + 9. 

It only remains to show that Player B outputs the same Xi with probability > 1 — e. We start 
with the following claim. 

Claim 4.2. For each n, P[/c > n] < e"". 
Proof. For each n, we have 

V[k > n] =P[ai iV ioii = 1, . . . , n • |ZY|] = (1 - 1/|ZY|)I"I-" < e"". 

□ 

Thus the probability that the binary encoding of k exceeds 1 + [log log l/e] bits is less than 
^_2-2r'°siogi/ei ^ ^^2^ remains to analyze StepHlof the protocol. We say that an element a = {x,p) 
survives iteration t if a G 2* ■ Q and it satisfies hj{x) = hj{xi) for all j = 1, . . . , for this t. 

Note that the "correct" element survives iteration t if and only if 2*^ > P{xi)/Q{xi). 

Claim 4.3. Let Ea^ he the event that the element selected by player A is ai, which is the i-th 
element on the tape. Denote k := \i/\l/(\'\. Conditioned on E^^, the probability that a different 
element aj with j £ {{k — 1) • \l/(\ + 1, . . . , k • \U\} .survives iteration t is bounded by e/2^~^^ . 

Proof. Without loss of generality we can assume that > 2, since for a singleton universe our 
sampling protocol will succeed trivially. This implies that for a uniformly selected a £ A, P[a ^ 
V] > 1/2, so: 

P[a eCfQ\ a^r]<P[aeCf Q]/P[a ^ V] < 2 ■ P[a e Ct ■ Q] < 2Ct/\U\. 

Denote K := k-\U\. Conditioning on the elements aj^_^K^^i, . . . , a^-i are distributed uniformly 
on ^ \ "P, and Oj+i , . . . , ax are distributed uniformly on A. For any such j = K — + 1, . . . ,i — 1, 
and for any C > 0, 

F[aj €C-Q]< 2C/\U\. 
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For such a j, surviving round t means aj belonging to 2* ■ Q and agreeing with Oj on st = 
1 + [log + (t + 1)^ random hashes /ii, . . . , /i^j . The probability of this event is thus bounded by 

P[aj survives round t] < P[aj G 2* ■ Q] ■ l'"' 

< 2 • 2*' • 2-^71^1 

< 2^+*'-^7|ZY| 

< 2-2t-ie/|ZY|. 

By taking a union bound over all j = K + . . , K, j / i, we obtain the bound of e/22*+i < 
e/2*+^ □ 

Thus for any Ea , the probability of Player B to output anything other than Xi conditioned on Ea 

□ 



To get a bound on the expected amount of communication in the protocol, as in Theorem 12.1 
note that 



E 



log P{x,)/Q{xi) + 2 + 5^1ogP(xi)/Q(xi)J = D (P| |Q) + 9 + 5 • E^^^py/log P{xi)/Q{x,) 
< D (P||Q) + 2 + 5 • JE,,^plogP(x,)/Q(xi) = D (P||Q) + 0(D + 1), 



where the inequality is by the concavity of This completes the proof. 

Remark 4.4. Note that if the parties are trying to sample many independent samples from distri- 
butions Pi, P2, ■ ■ ■ , with the receiving party knowing Qi, . . . , as in the setting of the Slepian-Wolf 
theorem, the analysis of the above protocol can easily be strengthened to show that the commu- 
nication is the right amount with high probability. This is because the central limit theorem can 
be used to show that samples Xi with higher than expected Pi{xi) / Qi{xi) are rare and do not 
contribute much to the communication on average (see for example Section [6. 2p . 

Remark 4.5. The sampling in the proof of Theorem 14.11 may take significantly more than one 
round. In fact, the expected number of rounds is 0(y^D [P\\Q)). We suspect that the dependence 
of the number of rounds in the simulation on the divergence cannot be eliminated, since D (P||(5) 
is not known to the players ahead of time, and the only way to "discover" it (and thus to estimate 
the amount of communication necessary to perform the sampling task) is through interactive com- 
munication. By increasing the expected communication by a constant multiplicative factor, it is 
possible to decrease the expected number of rounds to 0(logD {P\\Q)). 



5 Correlated Pointer Jumping 

Here we define the correlated pointer jumping problem, that is at the heart of several of our results. 
The input in this problem is a rooted tree such that 

• Each non-leaf node is owned by Player A or by Player B. 
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• Each non-leaf node owned by a particular player has a set of children that are owned by the 
other player. Each of these children is labeled by a binary string, in such a way that this 
coding is prefix free: no child has a label that is a prefix of another child. 

• Each node v is associated with two distributions on its children: childy known to Player A 
and childy known to Player B. 

• The leaves of the tree are labeled by output values. 

The number of rounds in the instance is the depth of the tree. 

The goal of the problem is for the players to sample the leaf according to the distribution 
that is obtained by sampling each child according to the distribution specified by the owner of the 
parent. We call the distribution of this path, the correct distribution. We give a way to measure 
the correlation between the knowledge of the two parties in the problem. 

For every non-root vertex w in the tree whose parent is v, define the divergence cost of w as 



The divergence cost of the root is set to 0. 

Given a path T that goes from the root to a leaf in the tree, the divergence cost of the path, 
denoted D (T) is the sum of the divergence costs of the nodes encountered on this path. Finally, 
the divergence cost of the instance F, denoted D (F) is the expected sum of divergence costs of the 
vertices encountered in the correct distribution on paths. 

We can use our sampling lemma to solve the correlated pointer jumping problem, with commu- 
nication bounded by the divergence cost: 

Theorem 5.1. There is a protocol that when given a k-round correlated pointer jumping instance 
F , can sample a path T such that there is an event E, with ^[E] > 1 — ke, and conditioned on E, 

• the parties both output the same sampled path T that has the correct distribution 

• the communication in the protocol is bounded by D (T) + 2A;log(l/e) + Sy^/cD (T) + 9A;. 

Proof. The protocol for sampling the path is obtained simply by repeatedly running the protocol 
from Theorem 14.11 In each step, the parties sample the correct child. For each round i let Ei denote 
the event that the parties are consistent after round i. When Ei occurs, the sampled vertex has 
the correct distribution, and Pr[ii^j] > 1 — e. Define E to be the intersection of the events Ei. Then 
Pr[£^] > 1 — ke. Conditioned on E, the sampled path has the correct distribution. Moreover, the 
if the sampled path is T = vqjVi, . . . ,Vk, then by Theorem 14. H the communication in the protocol 
is at most 
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k 

^ (d {vi) + \og{l/e) + log log(l/e) + 5 v/D(^ + o) 

i=l 

k 

< ^ (D {vi)) + 2A: log(l/e) + 5, A; • D + 9A; 

1=1 \ 1=1 

= D (T) + 2/c log(l/e) + b^Jk■T>{T) + 9k, 
where the inequahty is by the Cauchy-Schwartz inequahty. 

□ 

A key fact is that both the internal and external information cost of a protocol can be used 
to bound the expected divergence cost of an associated distribution on correlated pointer jumping 
instances. Since, in this work, we only require the connection to internal information cost, we shall 
restrict our attention to it. 

Given a public coin protocol with inputs X, Y and public randomness R, for every fixing of 
x,y,r, we obtain an instance of correlated pointer jumping. The tree is the same as the protocol 
tree with public randomness r. If a node v at depth d is owned by Player A, let M be the 
random variable denoting the child of v that is picked. Then define child^ so that it has the 
same distribution as M\ X = x,TT(X,Y)<:d = rv, and child^^ so it has the same distribution as 
M| Y = y,Tr{X,Y)<:d = rv. We denote this instance of correlated sampling by FTj{x,y,r). Let /i 
denote the distribution on X,Y. Next we relate the average divergence cost of this instance to the 
internal information cost of vr: 

Lemma 5.2. Ex,y,i? [D y, r))] = \C^{it) 

Proof. We shall prove that for every r, Ex,y [D {FT^{x,y,r))] = IC'^(7r,.). The proof can then be 
completed by Lemma [3. 131 

So without loss of generality, assume that vr is a private coin protocol, and write F{x, y) to 
denote the corresponding divergence cost. We proceed by induction on the depth of the protocol 
tree of vr. If the depth is 0, then both quantities are 0. For the inductive step, without loss of 
generality, assume that Player A owns the root node v of the protocol. Let M denote the child of 
the root that is sampled during the protocol, and let F{x, y)m denote the divergence cost of the 
subtree rooted at m. Then 



E [D(F(x,y))] 



E 

x,y,me^X,Y,M 



log(child^''(m)/childf^(?n)) + D(F(x,y), 



(1) 



Since for every x,y, M\xy has the same distribution as M\x, Proposition 13. 71 gives that the first 
term in Equationfflis exactly equal to /(X; M\Y) = I{X; M\Y) + I{Y; M\X). The second term is 
Em [Ex,y |m [D iF{X, Y) m)]] ■ For each fixing of M = m, the inductive hypothesis shows that the 
inner expectation is equal to I{X] n^X, Y)\Ym) + I{Y; tt{X, Y)\Xm). Together, these two bounds 
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imply that 



E [T>{F{x,y))] 

X,Y 

= I{X; M\Y) + I{Y; M\X) + /(X; tt{X, Y)\YM) + I{Y; tt{X, Y)\XM) 

□ 



6 Applications 

In this section, we use Theorem 15.11 to prove a few results about compression and direct sums. 

6.1 Compression and Direct sum for bounded-round protocols 

Here we prove our result about compressing bounded round protocols (Corollarv 12. 2p . We shall 
need the following lemma. 

Lemma 6.1. 

Proof of Corollary \2.S[ The proof follows by applying our sampling procedure to the correlated 
pointer jumping instance F.,^{x,y,r). For each fixing of x,y,r, define the event Gx,y^r to be the 
event E from Theorem 15.11 Then we have that P[G'] > 1 — ke. Conditioned on G, we sample from 
exactly the right distribution, and the expected communication of the protocol is 



E 



D y, R)) + 2k log(l/e) + 0{^/kT) {F^{X,Y,R)) + k) 



< E [D(F^(X,y,i?))] + 2A:log(l/e) + (/ E [kT> {F^{X,Y,R))] + k] , 

X.Y,R \\J X,Y,R I 

where the inequality follows from the concavity of the square root function. By Lemma 15.21 this 
proves that the expected communication conditioned on G is IC'^(7r)+2A; log(l/e)+0 (^y^klO^Jjr) + k 

□ 

6.2 Information = amortized communication 

In this section we will show that Theorem 15.11 reveals a tight connection between the amount 
of information that has to be revealed by a protocol computing a function / and the amortized 
communication complexity of computing many copies of /. Recall that IC'^(/, p) denotes the 
smallest possible internal information cost of any protocol computing / with probability of failure 
at most p when the inputs are drawn from the distribution fi. Observe that IC'^(/, p) is an infimum 
over all possible protocols and may not be achievable by any individual protocol. It is also clear 
that \C'fj,{f,p) may only increase as p decreases. 
We first make the following simple observation. 



Claim 6.2. For each f , p and p, 



limlCV(/,a) = ICV(/,/9) 

a— >-p 
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Proof. The idea is that if we have any protocol with internal information cost /, error 5 and input 
length i, for every e we can decrease the error to (1 — e)5 at the cost of increasing the information 
by at most e • i just by using public randomness to run the original protocol with probability 1 — e, 
and with probability e, run the trivial protocol where the players simply exchange their inputs. 
Thus as a tends to p, the information cost of the best protocols must tend to each other. □ 

Next we define the amortized communication complexity of /. We define it to be the cost of 
computing n copies of / with error p in each coordinate, divided by n. Note that computing n 
copies of / with error p in each coordinate is in general an easier task than computing n copies 
of / with probability of success 1 — p for all copies. We use the notation Dp'"'{f) to denote the 
communication complexity for this task, when the inputs for each coordinate are sampled according 
to p. Dp'^{f) was formally defined in Definition 13.161 

It is trivial to see in this case that Dp'"'{f) < n ■ Dp{f). The amortized communication com- 
plexity of / with respect to p is the limit 

AC(/^^) := hm Dli'^{f)/n, 

when the limit exists. We prove an exact equality between amortized communication complexity 
and the information cost: 

Theorem 6.3. 

AC(/^^) = IC^,(/,p). 
Proof. There are two directions in the proof: 

AC(/p) > \Cp{f,p). This is a direct consequence of Theorem 13.171 

AC(/^) < \Cp{f,p). Let 6 > 0. We will show that i:)^'"(/)/n < \C^{f,p) + <5 for all sufficiently 
large n. 

By Claim [UT^ there is an a < p such that IC'^(/, a) < \C'p{f,p) + 6/4. Thus there is a protocol 
TT that computes / with error < a with respect to p and that has an internal information cost 
bounded by I := \C^{f, p) + 6 /A. 

For every n, denote by vr"' the protocol that takes n pairs of inputs X",y" and executes in 
parallel, sending the first bits of each copy in the first round, and then the second bits in the second 
round and so on. Thus vr" has CC(7r) rounds, and communication complexity nCC(7r). Further, vr"' 
computes n copies of / as per Definition 13.161 with error bounded by a. 

We shall obtain our results by compressing tt"^. 

Let *7r denote the transcript of the z'th copy, and let Xi.,Yi denote the z'th inputs. Then 
observe that for all i, (Xj, 1^,* vr) are mutually independent of each other. Indeed, this implies that 
IC'^n(7r") = IC'^(*7r) = nlC'^(7r). On the other hand, compressing vr" incurs a per round 

overhead that is still dependent only on CC(7r). 

Let T" denote the random variable of the path sampled in vr". Let Ti,...,T„ denote the 
random variables of the n paths sampled in the individual copies of vr. Then, since each protocol 
runs independently, E [D (T")] = Yll=i E [D iT)]. Indeed, each vertex in the protocol tree of vr" 
corresponds to an n-tuple of vertices of vr, and if w corresponds to the vertices Q'w, . . . with 
parents v = {^v, . . . ," v) owned by Player A, then 
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By Lemma 15.21 E [D (T)] = IC'^(7r). Thus, by the central hmit theorem, for n large enough, 

Pr[D (T") > n • (ICV(7r) + 5/4)] < {p - a)/2. 

We use Theorem 15.11 to simulate vr", with error parameter e = (p — a)/2 and truncate the 
protocol after 

n • (ICV.(vr) + 5/ A) + 5^JcC{tt) ■ n ■ {\C ^{tt) + 5/4) + 2 log(l/e) + 9 • CC(^) 

bits of communication. The new protocol thus has error <a + p — a = p. On the other hand, for 
n large enough, the per copy communication of this protocol is at most IC'^(7r) + 6/2 as required. 

□ 

6.3 A complete problem for direct sum 

Let /" denote the function mapping n inputs to n outputs according to /. We will show that the 
promise version of the correlated pointer jumping problem is complete for direct sum. In other 
words, if near-optimal protocols for correlated pointer jumping exist, then direct sum holds for all 
promise problems. On the other hand, if there are no near-optimal protocols for correlated pointer 
jumping, then direct sum fails to hold, with the problem itself as the counterexample. Thus any 
proof of direct sum for randomized communication complexity must give (or at least demonstrate 
existence) of near-optimal protocols for the problem. 
We define the CPJ(C, I) promise problem as follows. 

Definition 6.4. The CPJ(C, /) is a promise problem, where the players are provided with a binary 
instance!! F of a C-round pointer jumping problem, i.e. player A is provided with the distributions 
child(f)^ and player B is provided with the distributions c\r\\\d{v)y for each v, with the following 
additional guarantees: 

• the divergence cost D (F) < I; 

• let pf be the correct distribution on the leafs of F; each leaf z of F are labeled with £(z) S 
{0, 1} so that there is a value g = g{F) such that 'Pz^_^^p [£(z) = g{F)\ > 1 — e, for some small 
£. The goal of the players is to output g{F) with probability > 1 — 2e. 

Note that players who know how to sample from F can easily solve the CPJ problem. It follows 
from |BBCR10| that: 

Tiieorem 6.5. If CP J{C, I) has a randomized protocol that uses T{C,I) := R(CPJ(C, /)) commu- 
nication, so that T{C,C/n) < C/k{n), then for each f , 

R(r) = n{k{n) . R(/)). 

In [BBClRinj a bound of T{C,I) = d{VC~I) is shown, which implies R(/") = ■ R(/)) 

for any /. Using Theorem 15. II we are able to prove the converse direction. 

Theorem 6.6. For any C > I > 0, set n := [C/I\, then 

R(CPJ(C,/)") = 0(Clog(nC/e)). 

^Each vertex has degree 2. 
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Thus, if there are parameters C and n such that CPJ(C, C /n) cannot be solved using I = C /n 
communication, i.e. T{C,C/n) > C/k{n) ^ C/n, then CPJ(C, C/n) is a counterexample to direct 
sum, i.e. 

R(CPJ(C,/)") = 0(C\ognC/e) = d{C) = d{k{n)R{C?]{C,C /n))) = o(n • R(CPJ(C, C/?i))). 

Proof, (of Theorem l6.6|) We solve CPJ(C, /)" by taking m := nlogn copies of the CPJ(C, /) problem 
representing log n copies of each of the n instances. The players will compute all the copies in parallel 
with error < 2e, and then take a majority of the logn copies for each instance. For a sufficiently 
large n this guarantees the correct answer for all n instances except with probability < e. Thus our 
goal is to simulate m copies of CPJ(C, /). We view CPJ(C, I)"^ as a degree-2"*, C-round correlated 
pointer jumping problem in the natural way. Each node represents a vector V = (fi, . . . of 
m nodes in the m copies of CPJ(C, I). The children of V are the 2™ possible combinations of 
children of {vi^ . . . ,Vm}- The distribution on the children is the product distribution induced by 
the distributions in ui, . . . , Vm- We claim that 

m 

D(CPJ(C,/);;^,...,,„) =^D(CPJ(C,/X,). (2) 

i=l 

This follows easily by induction on the tree, since the distribution on each node is a product 
distribution, and for each independent pairs (Pi, Qi), . . . , {Pm-, Qm) we have 

D (Pi X P2 X . . . X P^IIQl X Q2 X . . . X Qrn) = D (PlIIQl) + . . . + D {PmWQm) , 

by Lemma [3. 8 1 By applying ([2]) to the root of the tree we see that D (CPJ(C, J)™") < m-I < Clogn. 
Thus Theorem 15.11 implies that CP^{C, I)"" can be solved with an additional error of e/2 using an 
expected 

C log n + C log C/e + o(C log n) 
bits of communication. □ 
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